HP InfiniBand solution for 0 racle Application Server 




0 verview 

0 racle Application Server overview 

Advantages of InfiniBand in an 0 racle Application Server environment 

HP InfiniBand solution for 0 racle Application Server 

Solution overview 

Sample HP InfiniBand configuration for Oracle AS 10G and Oracle RDBMS 10G 

Installation process 

0 racle AS on InfiniBand installation process 

Hardware setup and configuration 

Red Hat Enterprise Linux® installation 

Installation of InfiniBand software components 

Oracle RDBMS 10G installation 

0 racle Application Server 10G installation 

Transparent SDP configuration 

Sample J 2 EE application installation 



0 verview 



This whitepaper explains how to implement an 0 racle Applications Server (AS) J2EE solution for the 
database connection using HP ProLiant servers, HP StorageW orks, and HP InfiniBand™ options. The 
whitepaper is intended for system administrators, system architects, and systems integrators who are 
considering the advantages of HP InfiniBand-based systems in an 0 racle AS J 2 EE environment. 

0 racle Application Server overview 

O racle AS is a fully integrated application server - including a J2EE Compatible container, an 
enterprise portal, wireless, business intelligence, single sign-on using a LDAP compliant directory, and 
much more. It is the integrated nature of 0 racle AS that distinguishes it in the marketplace. 0 racle 
strives notonly to provide a container for J2EE applications, but to enable J2EE applications to be 
managed more easily, deployed as web services, accessed via a single sign-on, made more secure, 
and tied more closely to business level activities. 

Typically, the 0 racle AS is installed on its own server or set of servers and these servers access an 
0 racle database instance on another server across a network. The most typical network connection 
between 0 racle AS server and the 0 racle database server is via TCP/ IP over Ethernet. 

A typical 0 racle Application Server system has the following components: 

• 0 ne or more application servers 

• 0 racle database instance 

• Public local area network (LAN ) 

Advantages of InfiniBand in an Oracle Application Server environment 

The most common method of communication between application server machines and database 
server machines today is TCP/ IP over Ethernet. TCP/ IP and Ethernet both have limitations. The TCP/ IP 
processing on the application server and database server machines can result in substantial CPU 
utilization on both machines with typical latencies of 50 to 100 microseconds (\is). Additionally, 
observed G igabit Ethernet throughput is limited to approximately 0.8 gigabits per second (G bps) on 
a single network interface card (N IC). 

InfiniBand, a standards-based alternative to Ethernet, overcomes the performance limitations of 
TCP/ IP-based solutions. The key advantages of InfiniBand over TCP/ IP and Ethernet is InfiniBand's 
high throughput, low CPU utilization, and ultra-low latency. The current InfiniBand bandwidth of 10 
G bps 1 provides observed performance over 8 G bps. CPU utilization of only 1-3 % with latency less 
than 5 \is is possible. 

InfiniBand supports its own InfiniBand-specific protocol, sockets direct protocol (SDP). This is the 
standard wire protocol for the InfiniBand Architecture (IBA) for support of stream sockets 
(SO CK_STREAM ) networking over IBA. 

There are two different versions of SDP protocol that are supported by O racle. The first is native SDP 
support, where the O racle product implements the SDP directly. The second protocol, called 
Transparent SDP, is where the O racle product is configured to use TCP, and the SDP protocol 
provider converts TCP connections to SDP without any configuration changes at the O racle level. 
N ative SDP provides support for asynchronous 1/ O , whereas Transparent SDP only supports 
synchronous 1/ O . 

Because it supports asynchronous 1/ O , native SDP supports one of the most important features of 
InfiniBand - remote direct memory access (RDM A). RDM A is a communications technique that allows 
data to be transmitted from the memory of one computer to the memory of another computer without 



1 InfiniBand 10G bps bandwidth is represented as 4x or four lanes of traffic each at 2.5 G bps. 
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passing through either device's CPU, without needing extensive buffering, and without calling to an 
operating system kernel. RDM A can bring about a substantial reduction in CPU utilization on both the 
application and database servers. 

Oracle's 10G RDBMS product supports native SDP; however, Oracle products that use an earlier 
version of the 0 racle DB client can only utilize Transparent SDP. Both 0 racle AS 9i and 10G Release 
1 use the 0 racle 9i DB client. Therefore, they can only use Transparent SDP. W hile this means that 
Oracle AS cannottake advantage of asynchronous I/O or RDMA today, it still benefits from the 
additional bandwidth and higher throughput provided by InfiniBand with Transparent SDP. 0 racle AS 
10G Release 2 will support native SDP. 

Applications that connect to the database by using thin JDBC driver are also restricted to Transparent 
SDP. Applications that use the OCI driver can use native SDP protocol provided the product support 
SDP natively. 

HP InfiniBand solution for 0 racle Application Server 

Solution overview 

M any HP customers run J2EE applications in an 0 racle Application Server environment using 
standards-based HP ProLiant servers and HP StorageW orks Fibre Channel connectivity products. 
W ith the addition of HP InfiniBand options for ProLiant servers, customers can implement a complete 
HP-based 0 racle Application Server solution using InfiniBand for the database connection. 

Figure 1 depicts a sample configuration composed of the following components: 

• Application Server 

- HP ProLiant DL380 G4 server 

- (4) 36GB disk drives 

• Database Server 

- HP ProLiant DL380 G4 server 

- (2) 36GB disk drives 

- HP StorageW orks solution 

(5HP StorageW orks M odular Smart Array (M SA) or Enterprise Virtual Array (EVA) 
BHP StorageW orks FCA2214DC Fiber Channel HBA 
(5HP StorageW orks SAN Switch 

• HP InfiniBand cluster interconnect 

- HP NC570C dual portPCI-X InfiniBand HCA 

- HP 24 port4x Copper Fabric Switch 
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Figure LHP InfiniBand solution for 0 racle AS high level diagram 
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Sample HP InfiniBand configuration for Oracle AS 10G and Oracle RDBMS 10G 
The orderable components necessary to build a 2-node sample configuration based on the HP 
ProLiant DL380 server, HP StorageW orks MSA 1000 for database storage, and HP ProLiant 
InfiniBand options for the private interconnect is shown in Table 1. 

Table 1. Sample configuration bill of materials 



Quantity Part Number 


Description 


HP ProLiant Server Nodes 


2 361011-001 


DL380R04 X3.6-1M B/ 800, 2GB High Performance 


2 311584-B21 


Intel® Xeon™ X3.6-1M B/ 800M Hz Processor O ption Kit 


4 343056-B21 


2GB of Advanced PC 2 PC3200 DDR2 SDRAM DIM M Memory Kit 
(2 x 1024 MB) 


6 286776-B22 


36.4GB 15,000 rpm, U320 Universal Hard Drive, 1" (internal 
server storage) 


HP InfiniBand Private Interconnect 


2 376158-B21 


HP NC570C PCI-X Dual-port4x Fabric Adapter 


2 376232-B21 


HP lm 4x Fabric Copper Cable 


1 376227-B21 


HP 24-Port4x Fabric Copper Switch 



HP StorageWorks Database Storage 
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Quantity 


Part Number 


Description 


i 
j. 


JL lO J J U£. J. 


StnranpW nrk<; ? fl h Dual Pnrt fid-Rit/ 1 3 3 M H 7 Pf l-X-tn-Fihrp 
jlu i a y cvv ui i\j l vjU; uiia i ruii, ut dii/ i j j i w i n r l i a u nuic 

Channel Host Bus Adapter for W 2K, W indows Server 2003 and 
Linux® 


1 


221692-B22 


5 Meter FC LC-LC M ulti-M ode cable 


1 


201723-B22 


Modular SAN Array 1000 with 256 M B cache 


1 


190211-001 


StorageW orks Enclosure M odel 4354R - Rack 


14 


286776-B22 


36.4GB 15,000 rpm, U320 Universal Hard Drive, 1" (shared 
storage) 



Installation process 

0 racle AS on InfiniBand installation process 

The following is a list of the installation steps required to implement O racle AS on InfiniBand using 
ProLiant servers. 

Hardware setup and configuration 

Follow the appropriate installation guides to prepare your site and rack mount the HP StorageW orks 
storage device, HP StorageW orks Fibre Channel switch, HP ProLiant servers, and HP 24 port 4x 
InfiniBand switch. See HP StorageW orks - Entry-level Storage Products , 

http:/ / hl8006.wwwl.hp.com/ storage/ entrvstorage.html , for the StorageW orks M SA 1000 storage 
device and related switches. See HP.com - HP ProLiant InfiniBand™ , 

http:/ / hl8004.wwwl.hp.com/ products/ servers/ networking/ index-ib.html , for the InfiniBand switch. 
See HP.com - ProLiant servers - Industry standard servers , 

http:/ / hl8004.wwwl.hp.com/ products/ servers/ platforms/ , for information on ProLiant servers. Use 
the HP FC HBA installation guide to install the Fibre Channel HBA in the database server and connect 
the Fibre Channel cables between the storage device, Fibre Channel switch, and the Fibre Channel 
HBA. See HP Storage SAN Infrastructure Products , 

http:// hl8006.wwwl.hp.com/ storage/ saninfrastructure.html, for HBA. Use the HP InfiniBand 4x 
HCA install guide to install the HC A in each server and connect the InfiniBand cables to the InfiniBand 
switch and both HCAs. See, HP.com - HP ProLiant InfiniBand™ , 

http:/ / hl8004.wwwl.hp.com/ products/ servers/ networking/ index-ib.html for InfiniBand. 

Configure the HP StorageW orks storage device for use with 0 racle 10G database using the HP 
StorageW orks configuration guide for the storage device you have chosen. 

Red Hat Enterprise Linux® installation 

To install Linux, follow the Red Hat Enterprise Linux installation guides for both application and 
database servers and apply any patches deemed necessary. HP also provides a Linux installation 
guide specifically designed for installing Linux on ProLiant servers at the following link. 

http://h200006.www2.hp.com/ be/ docs/ support/ SupportM anual/ c002 5 7 687/ c0025 7687.pdf 
Installation of InfiniBand software components 

Follow the instructions in the Install and configure HP InfiniBand 4x HCA Drivers for Linux section of 
the HP 4x InfiniBand HCA User G uide. By default, the drivers will automatically install support for all 
of the protocols used by 0 racle including SDP and Internet protocol over InfiniBand (IPolB). 
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0 nee the HP InfiniBand 4x HCA drivers have been installed and configured on both servers, modify 
one of the unused Linux Ethernet configuration files (or make a copy) for use as the private 
interconnect. Edit the hosts file to include both "hostnameJB" and the IP address defined in the Linux 
Ethernet configuration files used for private interconnect. 

Specify the private interconnect IP address and hostnameJB 

Both servers 

M odify the hosts file to include the node names and IP addresses used for the private interconnect. 

# vi /etc/hosts 

10.1.1.10 dbhostname_IB 
10.1.1.15 appshostname_IB 

Copy ifefg-etho and rename itifefg-ibo 

# cd /etc/sysconf ig/network-script s 

# cp ifefg-etho ifcfg-ibO 

Database server 

M odify if cfg-ibO to include only the items listed below. 

# vi /etc/sysconf ig/network-script s/ifcfg-ibO 

DEVICE=ibO 
BOOTPROTO=static 
IPADDR=10 .1.1.10 
NETMASK=2 55 .255.255.0 
ONBOOT=yes 

Application server 

M odify ifcfg-ibO to include only the items listed below. 

# vi /etc/sysconf ig/network-script s/ifcfg-ibO 

DEVICE=ibO 
BOOTPROTO=static 
IPADDR=10 .1.1.15 
NETMASK=2 55 .255.255.0 
ONBOOT=yes 

Oracle RDBMS 10G installation 

0 nee the HP InfiniBand 4x private interconnect is installed, configured, and tested, 0 raclelOG can 
be installed using the procedure documented in the 0 racle 10G installation guide, Installing 0 racle 
Database 10G on Linux x86. The following is a link for 0 racle latest installation guide. 

http:/ / www.oracle.com/ technology/ pub/ articles/ smiley lOqdb insta II. html 

Oracle Application Server 10G installation 

0 nee the database server is operational, 0 racle Application Server 10G can be installed using the 
procedures documented in the 0 racle installation guide, 0 racle Application Server 10G Installation 
G uide. The following is a link for 0 racle latest installation guide. 

http:/ / download-west.oracle.com/ docs/ cd/ B12428 13/ insta 11.9 04/ install/ toe. htm 
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Transparent SDP configuration 

The following steps are very crucial and require absolute path statements; any typing errors could 
render the system inoperable. The preload file, Id. so. preload points to a library, libsdp.so, to be 
loaded during system startup. If the path is incorrect, library name is misspelled, or library does not 
exist a reinstall of the operating system may be required. 

0 n the database server 

1. Verify the existence of / lib/ libsdp.so 

2. Stop the database 

3. Stop the listener 

4. # echo "I lib/ libsdp.so" > / etc/ Id. so. preload 

5. #echo "match listen * : 1 5 2 1 " »/ usr/ local/ topspin/ etc/ libsdp.conf 

6. #echo "match destination *:1521" »/ usr/ local/ topspin/ etc/ libsdp.conf 

7. Edit the listener.ora file to include the private interconnect settings. 

LISTEN ER = 

(DESCRIPTIO N_LIST = 
(DESCRIPTION = 
(ADDRESS_LIST = 

(ADDRESS = (PRO TO CO L = SDP) (HOST = dbhostna meJB) (PO RT = 1521)) 

) 

) 

N ET_ASYN C IO =FALSE 
SDP_ASYNC_IO=FALSE 

8. Edit the sqlnetora file to include the following lines. 
N ET_ASYN C IO =FALSE 
SDP_ASYNC_IO=FALSE 

9. Start the database 

10. Start the listener 

11. Check/ proc/ topspin/ sdp/ conn_main (should have 3 entries) 
0 n the application server 

1. Verify the existence of / lib/ libsdp.so 

2. #echo "/lib/ libsdp.so" >/ etc/ Id. so. preload 

3. #echo "match listen * : 1 5 2 1 " »/ usr/ local/ topspin/ etc/ libsdp.conf 

4. # echo "match destination * : 1 5 2 1 " » / usr/ local/ topspin/ etc/ libsdp.conf 

5. Edit the tnsnames.ora file to included the private interconnect settings. 
in t = 

(descripto in = 

(ADDRESS = (PRO TO CO L = TCP) (HO ST = dbhostna me_ IB) (PO RT = 1521)) 
(CONN ECT_DATA = 
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(SERVIC E_N AM E = INT) 

) 

) 

6. Check/ proc/ topspin/ sdp/ conn_main (should have 1 entries) 

7. Verify sqlplus connection with the database. 

Sample J 2EE application installation 

Installation notes and scripts are generally provided with the J2EE application software. 
Configure the J 2 EE application to utilize the private InfiniBand interconnect. 
M odify the data sources for the sample J2EE Application as in the following example: 

IASDBDS 

JDBC URL jdbc:oracle:thin:@dbhostname_IB:1521:INT 
Test the connectivity 

1. sqlplus <0 racle user name>/ <0 racle password >@<0 racle Service N ame> 

2. Browse to J2EE Application login screen 
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For more information 



For additional information, refer to the resources detailed below. 



Rpcourrp dp^r rintion 

1 \CjUUI V_v^ U CjLI 1 VJ LI \J 1 1 


W ph addrp^ 


iir^r^ i 1 ii r 1 i 

HP ProLiant InfniBand 


http :/ / hl8004.wwwl.hp.com/ products/ servers/ networkinq/ index- 


options 


ib.html 


HP StorageWorks 


http:/ / hl8006.wwwl.hp.com/ storaqe/ arravsvstems.html 


M nrl i ila r ^ma rt 




Arrays (MSA) 




HP ProLiant servers 


http:/ / hl8004.wwwl.hp.com/ products/ servers/ platforms/ 






HP Parallel Database 


http:/ / hl8004.wwwl.hp.com/ solutions/ enterprise/ hiqhavailability/ o 


Clusters (PDC) for 


racle/ index.html 


H rarlo DAP 
U Id Lie KAL 




HP support 


http:/ / welcome.hp.com/ country/ us/ en/ support.html 


documentation and 




software downloads 




Oracle Application 


http:/ / www.oracle.com/ appserver/ index.html 


Server 
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